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Executive Summary 


This evaluation study provides information about the implementation and outcomes of the 
21st Century Community Learning Centers (21st CCLC) program in West Virginia, from 
September 2015 through May 2016, in which 12,388 students participated overall. 


Method of study. The report draws on information from online surveys of directors of thirty- 
six 21st CCLC programs and from regular-day school teachers of 2,034 of the 5,163 
students who participated in programming for at least 30 days. It also draws on West 
Virginia General Summative Assessment (WVGSA) scores for 1,864 of the students who 
participated for at least 30 days, and a matching group of 1,864 nonparticipants used as a 
control. The students were grouped by grade level and compared in mathematics and 
English/language arts (ELA) using changes in scale score means for Grades 4—11. The 
mean performance level changes for the two groups were also compared, aggregated by 
programmatic level. 


Findings. Most participating students were in the elementary grades. The mean number of 
days students attended ranged from about 12 to 112 days by program. Teachers reported 
that about 60% of students improved their behavior in the regular classroom and nearly 70% 
improved in completing their homework and class participation. The quasi-experimental 
study using scale score means for Grades 4-11 showed no statistically significant 
differences between the 21st CCLC participants and nonparticipants, except for fourth 
graders in ELA, with participants having a slight edge on nonparticipants. The comparison 
by programmatic level showed a statistically significant higher gain in ELA among 
nonparticipant elementary students compared with participants and a significantly higher 
gain in mathematics performance levels among participating high school students compared 
with nonparticipating high school students. 


The largest sources of program volunteers were K-12 service learning programs, parents, 
and higher education service learning programs. The groups with which program directors 
reported working most successfully were community organizations, higher education service 
learning programs, and local businesses. Regarding work with partners, the most frequent 
types of support received were in programming, program resources, and evaluation. 


All program directors considered their various partnership activities to be at least moderately 
effective. Program directors reported their greatest need for more professional development 
to be in programming, for technical assistance in federal/state requirements; and for 
information resources in programming and collaboration. More than 60% of program 
directors reported at least moderate success in parent and community involvement, which is 
a large gain compared with less than 40% the previous year. In responses to open-ended 
questions, the most frequently mentioned successes were in the area of program 
improvement, most often-mentioned challenges were in staffing and staff development. 
Lastly, program directors were asked to make recommendations for how to improve the 
program for the future. Among program directors who commented, the most frequently 


vii 


mentioned recommendations were for more networking opportunities and sharing of best 
practices, and for additional improvements in the reporting systems. 


Limitations of study. We cannot assume that the 21st CCLC attendance was a key factor in 
the improvement of behaviors perceived by teachers. Some results are based on 
perceptions of teachers and program directors. 


vill 


Introduction 


The West Virginia Department of Education (WVDE) administers the 21st Century Community 
Learning Centers (21st CCLC) to provide opportunities for communities to establish or expand 
activities in communities that 


1. provide opportunities for academic enrichment, including providing tutorial services to 
help students—particularly students who attend low-performing schools—to meet state 
and local student academic achievement standards in core academic subjects, such as 
reading and mathematics; 


2. offer students a broad array of additional services, programs, and activities—such as 
youth development activities; drug and violence prevention programs; counseling 
programs; art, music, and recreation programs; technology education programs; and 
character education programs—that are designed to reinforce and complement the 
regular academic program of participating students; and 


3. Offer families of participating students opportunities for literacy and related educational 
development. 


The 21st CCLC program was authorized under Title IV, Part B, of the Elementary and 
Secondary Education Act, as amended by the No Child Left Behind Act of 2001, which 
transferred administration of the program from the U. S. Department of Education to state 
education agencies. ' 


WVDE makes competitive local grants based on available federal funding to eligible 
organizations to support the implementation of community learning centers that will aid student 
learning and development. Eligible applicants are public and private agencies, city and county 
governmental agencies, faith-based organizations, institutions of higher education, and for-profit 
corporations. 


The purpose of this evaluation study is to provide information about the implementation and 
outcomes of the 21st CCLC program in West Virginia, during the period from September 2015 
through May 2016. 


Evaluation Questions 
This evaluation study addresses several broad evaluation questions: 


EQ1 Student participation and impacts. Which students were referred to 21st CCLC, at what 
levels of participation, and to what effect? 


EQ2 Volunteers and partnerships. How did programs operate with regard to volunteers, 
partnerships, and information sharing? 


1 The Every Student Succeeds Act, signed into law Dec. 20, 2015, also authorizes the 21st CCLC 
program; however, its provisions do not take full effect until the 2017-2018 school year. 


Methods 


EQ3 Professional development and technical assistance. How well did professional 
development and technical assistance support 21st CCLC programs, which formats are 
preferred, and what topics are most needed? 


EQ4 Parent and community involvement. What was the level of success in involving parents 
and community members? 


EQ5 Improvement and accountability processes. How helpful to 21st CCLC programs were 
improvement and accountability processes? 


EQ6 Successes, challenges, and recommendations. What do program directors view as their 
major successes, challenges, and recommendations for the future of the program? 


Methods 


We addressed EQ1 using responses to the 21st Century Community Learning Centers (21st 
CCLC) Teacher Survey from classroom teachers of students who had participated in a 21st 
CCLC program for at least 30 days during the regular school year and whose parents had 
provided consent. Teachers’ responses were analyzed using descriptive statistics. We 
determined levels of participation in each program based on 21st CCLC entry of daily student 
attendance in the 21st CCLC database; these data were also used to determine the number of 
days of participation (dose strength) for individual students. Using quasi-experimental methods 
we tested impacts on West Virginia General Summative Assessment (WVGSA) scores provided 
by the WVDE for students who participated in a 21st CCLC program for at least 30 days. Data 
were collected from the 21st CCLC and West Virginia Education Information System (WVEIS) 
reporting systems. 


The remaining five evaluation questions (EQ2—EQ6) were addressed using responses from the 
21st CCLC Program Director Survey, which were analyzed using descriptive statistics. This 
survey questionnaire was streamlined in April 2015 to reduce the burden on program directors 
while still collecting sufficient data to adequately address each of the evaluation questions. 


Participants in the study included teachers of students who had attended the 21st CCLC 
program for at least 30 days, whose parents had not denied consent for their participation in the 
study. In the quasi-experimental analysis, we selected the student treatment group contingent 
upon their having participated in a 21st CCLC for at least 30 days; we also used scores from a 
matching control group of students not known to have been participants. The only other 
participants and subjects in the study were 21st CCLC program directors, all of whom were 
contacted to participate in the study. 


A more detailed description of the methods is included in Appendix A, page 17. Survey 
instruments are in Appendix B, page 20, and sample informed consent forms can be found in 
Appendix C, page 24. A brief summary of the methods and data sources used in this study can 
be found in Table 1. 


Table 1. 


Methods 


Summary of Evaluation Questions and Data Collection Methods 


Evaluation question 


Method of analysis/ 
data source 


Results reported 


EQ1. Student 
participation and 
impacts. Which 
students were referred 
to 21st CCLC, for what 
reasons, at what levels 
of participation, and to 
what effect? 


Descriptive statistics/ 
Online 21st CCLC 
Teacher Survey 


Among students who had participated in a 21st CCLC 
for at least 30 days, which behaviors did teachers 
report as having improved? 


Descriptive statistics/ 
WVDE 21% CCLC 
database 


What was the distribution of attendance by grade level? 


Across programs, what was the level of participation 
(dose strength)? 


Quasi-experimental 
study/General 
Summative Assessment 
scores and WVDE 21° 
CCLC database 


Among students who had participated in a 21st CCLC 
for at least 30 days, what was the impact of 21st CCLC 
participation on 2-year English/language arts (ELA) and 
mathematics gains? 

What was the impact of 21st CCLC participation on 
end-of-year ELA and mathematics achievement?* 
What were the year-to-year changes in ELA and 
mathematics achievement for each group (21st CCLC 
participants and nonparticipants) independently? 

Were ELA and mathematics gains experienced by 21st 
CCLC participants significantly different from those 
gains experienced by nonparticipants? 


EQ2. Volunteers and 
partnerships. How did 
programs operate with 
regard to volunteers 
and partnerships? 


Descriptive statistics/ 
Online 21st CCLC 
Program Director 
Survey 


How many volunteers were involved in programs and 
from which sources? 

At what level of success did program directors work 
with each source of volunteers? 

How many partners did programs work with, and what 
was the nature of partners’ support? 

How effective were collaborations with partners? 


EQ3. Professional 
development and 
technical assistance. 
How well did 
professional 
development and 
technical assistance 
support 21st CCLCs, 
which formats are 
preferred, and what 
topics are most 
needed? 


Descriptive statistics/ 
Online 21st CCLC 
Program Director 
Survey 


What was the quality of professional development 
offered by counties and RESAs on various topics; and 
what is the ongoing need for more on these topics from 
these sources? 

What was the quality of professional development 
offered by WVDE on various topics; and what is the 
ongoing need for more on these topics from this 
source? 

What was the quality of professional development 
offered by the U.S. Department of Education on various 
topics; and what is the ongoing need for more on these 
topics from this source? 

How helpful were various forms of technical 
assistance? 


EQ4. Parent and 
community involvement. 
What was the level of 
success in involving 
parents and community 
members? 


Descriptive statistics/ 
Online 21st CCLC 
Program Director 
Survey 


How successful were programs in involving parents, 
guardians, and community members? 

How many adults were involved in 21st CCLC activities 
and what was the nature of their involvement? 


Results 


Table 1. Summary of Evaluation Questions and Data Collection Methods 


Method of analysis/ 


Evaluation question data source Results reported 

EQ5. Improvement and Descriptive statistics/ How helpful was the continuous improvement process 
accountability Online 21st CCLC for after school (CIPAS)? 

processes. How helpful = Program Director How helpful were the WVDE monitoring visits? 

to 21st CCLCs were Survey 


improvement and 
accountability 


processes? 

EQ6. Successes, Descriptive statistics/ What did program directors view as their major 
challenges, and Online 21st CCLC successes? 

recommendations. Program Director What did program directors view as their major 
What do program Survey challenges? 


directors view as their 
major successes, 
challenges, and 
recommendations for 
the future of the 
program? 


What specific professional development or technical 
assistance topics did program directors think would be 
most helpful for WVDE staff to deliver during the 
upcoming school year? 

What recommendations did program directors have for 
improving the 21st CCLC program? 


Results 


Of the 12,387 students served by the 21st Century Community Learning Centers (21st CCLC), 
5,377 attended the program for 30 or more days during the regular school year. We received 
2,034 Teacher Survey responses that provided information about 30-or-more-day students. All 
27 program directors responded to the Program Director Survey; they provided separate 
responses for each grant held by their organization, resulting in 36 completed surveys. 


EQ1 Student Participation and Impacts 


Which students were referred to 21st CCLC, for what reasons, at what levels of participation, 
and to what effect? 


Student participation by grade level 


Most student participants—6,900 of 12,387 or 55.7% were in elementary school (Grades PK-5). 
An additional 3,239 or 26.2% were in middle school (Grades 6-8), and 2,246 or 18.1% were in 
high school (Grades 9-12) in 2015-2016. See Figure 1 for a breakdown by individual grade.? 


2 Grade levels for two students were coded incorrectly and were therefore excluded from these counts. 


Results 


Levels of participation (dose 
strength) PK-K (1 = 845) LE 79% 

. : Grade1(n=1,031) EE 8% 
Using data entered into the 21st Grade 2 (1-11) A 10% 


( 
CCLC database, we calculated the Grade 3(1=1,231) TT 10% 
average number of days attended per Grade 4(0=1321) a 11%, 
( 
( 


student and the percentage of @ Grade5 (1-122) TT 10%, 
enrollment that attended for 30 or eee ee ee ee a a 
more days for each program and the Beaded (\-11)-) 9; 
state overall (see Table 2). For the isda g (\- 832) a 79, 

state overall, the average number of Grade 9(n=690) A 6% 

days attended per student was 36.1. Grade 10(n=564) I 5% 

Average attendance rates for the Grade 11(n=505) ——~ 4% 

programs ranged from 11.8 (Boone Grade 12(n=487) | 4% 


County Public Schools) to 111.6 days 0% 2% 4% 6% 8% 10% 12% 
(Bob Burdette Center). As for the 
percentage of students enrolled who 
attended 30 or more days, the overall 
rate was 41.7%. Programs ranged 
from 10.7% (PATCH-Mason County) 
to 88.8% for the Bob Burdette Center. 


Percent of all student participants 


Figure 1. Percentage of Student Participation by Grade Level 


Data source: 21st CCLC database for 2015-2016 regular 
school year 


Table 2. Program Attendance Dose Strength 


Total Mean days Students Students 

enrollment Total days attended with 30+ with 30+ 

21st CCLC program (N) attended _ per student days (N) days (%) 
All 12,388 447,276 36.1 5,163 41.7 
Bob Burdette Center 188 20,983 111.6 167 88.8 
Marion County Public Schools 309 27,094 87.7 271 87.7 
World Vision 142 8,856 62.4 111 78.2 
Ritchie County Public Schools 216 11,542 53.4 164 75.9 
Salvation Army Boys and Girls Club 160 12,614 78.8 120 75.0 
Preston County Public Schools 481 26,682 55.5 354 73.6 
Partnership of African-American Churches 163 13,832 84.9 117 71.8 
Mountaineer Boys and Girls Club 278 20,079 72.2 186 66.9 
Lincoln County Public Schools 509 16,424 32.3 306 60.1 
Morgan County Public Schools 410 14,871 36.1 227 55.4 
Human Resource Development Foundation 135 8,339 61.8 70 51.9 
Boys and Girls Club, Eastern Panhandle 347 11,765 33.9 162 46.7 
Step-by-Step 272 9,131 33.6 126 46.3 
Clay Center 147 5,431 36.9 64 43.5 
RESA 7 637 21,310 33.5 272 42.7 
Wayne County Public Schools 2,709 103,782 38.3 1120 41.3 


Table 2 continues next page 


Results 


Table 2. Program Attendance Dose Strength 


Total Mean days Students Students 

enrollment ‘Total days attended with 30+ with 30+ 

21st CCLC program (N) attended _ per student days (N) days (%) 
McDowell County Public Schools 894 21,299 23.8 303 33.9 
Cabell County Public Schools 276 8,795 31.9 93 33.7 
PATCH-Jackson County 700 19,194 27.4 202 28.9 
PATCH-Ravenswood 319 7,885 24.7 85 26.6 
RESA 4 1,469 30,521 20.8 348 23.7 
RESA 2 193 3,341 17.3 44 22.8 
PATCH-Roane County 820 15,831 19.3 182 22.2 
Boone County Public Schools 289 3,420 11.8 33 11.4 
PATCH-Mason County 337 4,255 12.6 36 10.7 


Data source: 21st CCLC database 2015-2016 regular school year 
Student changes in behavior 


Teachers rated the degree of 
improvement of 2,034 students on 
two measures—homework 
completion and class participation, 
and student behavior—using the 
following scale: 3 = improvement, 2 = 
no change, and 7 = decline (see 
Appendix B, page 23 for survey 
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As shown in Figure 2, teachers 
reported nearly 60% of students 
improved their behavior since they 
began participating in the 21st CCLC 
program, and nearly 70% improved 
their homework completion and class 
participation. 


Student behavior Homework completion 
and class participation 


Category of student performance 


@Decline MNochange ®lmprovement 


Figure 2. Percentage of 21st CCLC Students by Behaviors 
Needing Improvement 

The second administration of a new Data source: 21st CCLC Teacher Survey May-June 2016 

online assessment, the West Virginia 

General Summative Assessment (WVGSA) took place in the spring of 2016, which allowed us 

to study gain scores for the first time. The test was administered to students in Grades 3-11 for 

both mathematics and English/language arts (ELA). 


Impact of student participation 


To study academic impacts, we conducted a quasi-experimental study comparing an 
experimental group of students who had participated in 21st CCLC programming for at least 30 
days and a control group of similar students not known to have participated in the program. The 
control group students were selected using propensity score matching (PSM) on specific criteria 


(see Appendix A, 
page 17 for more 
details of methods 
used). There were 
1,864 students in 
both the 
experimental (21st 
CCLC participant) 
and the control (non- 
participant) groups in 
Grades 4-11. 


Table 3 presents 
the results of 
independent 
samples t tests used 
to determine any 
statistically 
significant 
differences between 
21st CCLC 
participants and 
nonparticipants in 
ELA mean scale 
score declines/gains 
from 2015 to 2016 
on the WVGSA for 
each grade level. In 
no case were the 
slight differences 
observed statistically 
significant. 

We also examined 
differences between 
21st CCLC 
participants’ and 
nonparticipants, by 
programmatic 

level (Table 4). In 
this case, we used 
ELA performance 
levels instead of 


Results 


Table 3. Comparison of 1-Year ELA Point Declines/Gains for 21st 
CCLC Participants and Nonparticipant on WVGSA: 2015 to 
2016 Testing Years 


Mean Std. Std. Error Sig. (2- 

Grade Group N_ gain/loss Deviation Mean tailed) 

3to 4 Nonparticipant 592 41.03 56.810 2.335 387 
Participant 592 43.82 54.044 2.221 

4to5 Nonparticipant 456 47.94 53.633 2.512 .140 
Participant 456 42.83 50.795 2.379 

5to6 Nonparticipant 409 17.47 53.584 2.650 .697 
Participant 409 18.92 52.510 2.596 

6 to7 Nonparticipant 369 31.82 58.326 3.036 821 
Participant 369 32.76 54.766 2.851 

7to8 Nonparticipant 254 19.58 48.923 3.070 .558 
Participant 254 22.28 54.650 3.429 

8to9 Nonparticipant 92 -8.49 63.021 6.570 504 
Participant 92 -1.98 68.758 7.169 

9to 10 Nonparticipant 127 10.35 69.721 6.187 549 
Participant 127 5.29 64.688 5.740 

10 to 11 Nonparticipant 71 7.66 84.540 10.033 .064 
Participant 71 31.89 69.090 8.200 


Data source: WVEIS 2015 and 2016 WVGSA test scores 


Table 4. Comparison of 21st CCLC Participants’ and Nonparticipants' 
WVGSA ELA Performance Level 1-Year Declines/Gains by 
Programmatic Level: 2015 to 2016 


Performance Group 
Program level Number level decline/gain Nonparticipants Participants 
Elementary 2096 Decline 17.8%a 18.0%a 
No change 56.9%a 60.6%a 
Gain 25.3%a 21.4%p 
Middle 2064 Decline 18.0%a 17.7%a 
No change 61.2%a 61.5%a 
Gain 20.7%a 20.7%a 
High 580 Decline 21.4%a 16.6%a 
No change 56.9%a 57.6%a 
Gain 21.7%a 25.9%a 


Note: Highlighted cells show the only differences that were statistically significant. Data source: 
WVEIS 2015 and 2016 WVGSA performance levels 


Results 


scale scores, looking at the percentages of students who declined at least one performance 


level, stayed the same, 
or gained at least one 
performance level. We 
used a chi-square test 
to check the statistical 
significance of any 
differences between 
the two groups. At the 
elementary school level 
there was a statistically 
significant difference 
between 21st CCLC 
participants and 
nonparticipants with a 
higher percentage of 
nonparticipants (25.3%) 
gaining at least on 
performance level than 
participants (21.4%). 


The same tests were 
applied to 21st CCLC 
participant and non- 
participant math scores 
and performance 
levels. Table 5 presents 
the results of 
independent samples ft 
tests used to determine 
any statistically 
significant differences 
between 21st CCLC 
participants and 
nonparticipants in math 
scale scores. The only 
statistically significant 
differences among the 
groups was for Grades 
3 to 4, where 21st 
CCLC participants 
outgained 
nonparticipants. 


Table 5. Comparison of 1-Year Math Scale WVGSA Score Gains 
for 21st CCLC Participant and Nonparticipant: 2015 to 
2016 Testing Years 
Mean Std. Std. Error — Sig. (2- 
Grade Group N gain/loss Deviation Mean tailed) 
3to4  Nonparticipant 592 41.65 44.176 1.816 014 
Participant 592 47.83 42.433 1.744 
4to5  Nonparticipant 456 32.29 44.809 2.098 .243 
Participant 456 28.72 47.549 2.227 
5 to 6 Nonparticipant 409 11.69 55.933 2.766 .084 
Participant 409 4.77 58.598 2.897 
6 to 7 Nonparticipant 369 21.75 60.522 3.151 .216 
Participant 369 26.84 50.629 2.636 
7to8 Nonparticipant 254 18.16 68.045 4.270 .103 
Participant 254 20.30 57.870 3.631 
8to9 Nonparticipant 92 -19.63 84.342 8.793 .220 
Participant 92 -4.82 79.013 8.238 
9to10 Nonparticipant 127 16.78 86.230 7.652 -506 
Participant 127 10.05 74.300 6.593 
10 to 11 Nonparticipant 71 9.25 73.548 8.728 .295 
Participant 71 22.27 73.899 8.770 


*Highlighted cells show only statistically significant differences between the two groups (.05 
confidence level). Data source: WVEIS 2015 and 2016 WVGSA test scores 


Table 6. 


Comparison of 21st CCLC Participants’ and 


Nonparticipants' WVGSA Math Performance Level 1-Year 
Declines/Gains by Programmatic Level: 2015 to 2016 


Perf. level Group 
Program level Number decline/gain Nonparticipants Participants 
Elementary 2096 Decline 21.4%a 18.9%a 
No change 59.4%a 61.8%a 
Gain 19.2%a 19.3%a 
Middle 2064 Decline 16.3%a 14.1%a 
No change 65.8%a 68.9%a 
Gain 17.9%a 17.0%a 
High 580 Decline 19.7%a 16.9%a 
No change 69.0%a 65.5%a 
Gain 11.4%a 17.6%b 


Note: Highlighted cells show the only differences that were statistically significant. Data 
source: WVEIS 2015 and 2016 WVGSA performance levels 


Results 


Table 6 shows a statistically significant difference at the high school level, with more 21st CCLC 
participants (17.6%) experiencing at least a one-level gain in performance than nonparticipants 


(11.4%). 


EQ2 Volunteers and Partnerships 


How did programs operate with regard to volunteers and partnerships? 


Volunteers 


Program directors reported recruiting volunteers from several sources as shown in Table 7. In 


sheer numbers, K-12 
service learning 
programs were the 
largest source of 
volunteers (n = 744), 
followed by parents (n = 
565), and higher 
education students 
service learning 
programs (n = 526). As 
shown in Table 7, the 
groups of volunteers 
that program directors 
reported as the most 
successfully integrated 
into their programs 
included community 
organizations, higher 
education service 
learning students, local 
businesses, and faculty 
members. 


Partnerships 


Based on reports from 
program directors, 
programs engaged ina 
variety of functions with 
partners. The functions 
with the greatest 
number of engaged 
partners (see Table 8) 
were resources (292 


Table 7. Number of Volunteers Recruits by Source and Success 


Rate 

Total Percent using Level of 

volunteers this source success 

Community organizations 290 83.3 3.7 
Service learning (Higher educ. students) 526 58.3 3.5 
Local businesses 250 77.8 3.5 
Faculty members 275 69.4 3.4 
Service learning (K-12 students) 744 63.9 3.3 
AmeriCorps 117 50.0 3.2 
Local clubs (e.g. Kiwanis, Lions) 73 55.6 3.2 
Faith-based organizations 168 63.9 3.2 
Parents 565 86.1 3.1 
Senior volunteers 50 27.8 3.0 
Other 66 13.9 2.7 


NOTE: 36 of 36 program directors responded to this portion of the survey. 


Scale: 1 = no success, 2 = slight success, 3 = moderate success, and 4 = great 
success. Data source: 2016 21st CCLC Program Director Survey May-June 2016 


Table 8. Number and Percent of Partners by Function, Success 


Rate 

Total partners Percent of pro- 

engaged in grams with partners 
Function this activity in this activity Level of success 
Training 103 771 3.6 
Joint planning 189 80.0 3.5 
Resources 292 91.4 3.5 
Programming 220 97.1 3.4 
Management 89 57.1 3.4 
Funding 89 74.3 3.2 
Evaluation 63 82.9 3.2 
Other* 7 5.7 4.0 


NOTE: 36 of 36 program directors responded to this portion of the survey. 
SCALE: 1 = no success, 2 = slight success, 3 = moderate success, and 4 = great success. 
Data source: 2016 21st CCLC Program Director Survey May-June 2016 


Results 


partners; 91.4% programs with partners in this activity); programming (220 partners; 97.1% 
programs with partners in this activity), and joint planning (189 partners; 80% programs with 
partners in this activity). Program directors rated all activities engaged in with partners as at 
least moderately successful, with training, joint planning, and resources as having the highest 
level of success. 


EQ3 Professional Development and Technical Assistance 


How well did professional development and technical assistance support 21st CCLC programs, 
which formats are preferred, and what topics are most needed? 


Professional development 


As shown in Table 9, West Virginia program directors overall received the most training from the 
WVDE and the U.S. Department of Education (USED). The best attended trainings for all three 
sources combined in descending order were, programming (64), federal/state requirements 
(60), collaboration (55), and family involvement (55). 

Ratings of the effectiveness of the trainings were similar for all three sources (Table 9). The 
topics receiving highly effective ratings (>3.5 on a 4-point scale) included communications/ 
marketing (USED), staff development (USED), and federal/state requirements (WVDE). Two 


Table 9. Number of Program Directors Who Reported Attending Professional Development 
on This Topic and Mean Rating by Source of Professional Development 


All 
sources LEA/RESA WVDE USED 

Mean Mean Mean 

Topic Number Number rating Number rating Number rating 
All topics 89 23 3.1 33 3.2 33 3.2 
Programming 64 21 3.0 33 3.3 10 3.2 
Collaboration 55 21 3.4 29 3.2 5 3.4 
Communications/marketing 44 16 2.7 24 3.0 4 3.8 
Staff development 53 19 3.5 26 3.2 8 3.6 
Integrating afterschool w/ regular school day 54 16 3.1 28 3.1 10 2.4 
Project management 49 15 3.2 28 3.3 6 3:3 
Federal/state requirements 60 17 3.2 32 3.6 11 2.8 
Family involvement 55 16 2.9 28 2.8 11 2.5 
Program sustainability 43 10 2.7 28 2.6 5 3.0 
STEM/STEAM 52 16 3.5 31 3.5 5 3.4 
Program evaluation 45 14 3.1 26 3.4 5 3.4 
Policy and advocacy 38 12 2.8 20 3.0 6 3.2 


NOTE: Of the 36 Program Director Survey responses, 23 responded as having attended professional development 
provided by LEA/RESA sources; 33 responded as having attended professional development from WVDE and 
USED sources. 

SCALE: 1 = not effective, 2 = slightly effective, 3 = moderately effective, 4 = highly effective 

Data source: 2016 21st CCLC Program Director Survey May-June 2016 
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Results 


topics received mean ratings in the slightly effective range (S 2.5), both from USED: integrating 
afterschool with the regular school day and family involvement. All other topics from all other 
sources had mean ratings in the moderately effective range. 


Technical assistance 


Program directors reported that emails, oeaa 
phone calls/conference calls, and site Phone call/conference call 


visits were very helpful forms of = Site visit 
technical assistance (Figure 3). All other x) rae 

f . m4 Action plan feedback 
forms received average ratings of = Webinar 
moderately helpful. Peer learning teams 


7 Other 
Future needs for professional 


development, technical assistance, oo a 20 en ae 
and information resources Mean helpfulness rating 


Regarding professional development 


; Figure 3. Helpfulness of Various Types of Technical 
needed in the future, the three most 


Assistance 
requested topics were Plog ainmning; NOTE: Directors of all 36 programs responded to this 
program sustainability, and family portion of the 2016 21st CCLC Program Director Survey 
involvement (Figure 4). Program May-June 2016 
directors reported the greatest need for = SCALE: 1 = not helpful, 2 = slightly helpful, 3 = 
technical assistance in federal/state moderately helpful, 4 = very helpful 


requirements, collaboration, project 


Programming 

Federal/state requirements 
Collaboration 

Project management 
Integrating afterschool with the regular school day 
Staff development 
STEM/STEAM 

Program sustainability 
Family involvement 
Communications/marketing 
Program evaluation 


Policy and advocacy 


oO 
wn 


10 15 20 


N 
ea] 


Number of program directors 


m Professional development @ Technical assistance m Information resources 


Figure 4. Most-Needed Topics for Profession Development, Technical Assistance, and Information 
Resources 


Data source: 2016 21st CCLC Program Director Survey May-June 2016 
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management, integrating afterschool with the regular school day, and staff development. 
Relatively large numbers of program directors want information resources on programming, 
collaboration, federal state requirements, and STEM/STEAM. It is notable that for these four 
topics, more program directors indicate a need for information resources than for the other two 
forms of support (professional development and technical assistance). 


In addition to the multiple choice questions summarized above, program directors had the 
opportunity to respond to an open-ended question about their needs for professional 
development. Their responses are summarized below. 


Topics with at least six mentions included the following: 


Sustainability, sustainability plans, and fundraising - 19 mentions (e.g. grant reviewers’ 
expectations for sustainability plans, trends in future funding, development of 
endowments) 

Family engagement/involvement — eight mentions (e.g., editing home-learning packets, 
increasing attendance at family events) 

Data entry and reporting — seven mentions 

Collecting, interpreting, and analyzing academic data — six mentions 

Use of volunteers — six mentions (e.g., AmeriCorps, community volunteers) 


EQ4 Parent and Community Involvement 


What was the level of success in 
involving parents and community 
members? 


Program directors reported 
greater success involving parents 
and community members than in 
the previous year. In the 2015 
survey, about 60% of directors 
reported slight success at best; 
however, in 2016 a comparable 
percentage reported moderate or 
great success (Figure 5). 
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Figure 5. Level of Success Involving Parents and Community 


Data source: 21st CCLC Program Director Survey May-June 
2016 
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Program directors were also asked about Table 10. Parent and Community Member 

attendance of adults at four types of Participation by Type of Activity 

activities. As shown in Table 10, the Mean 
greatest level of participation—an average number of 
across the programs of 70 participants— [eugene 
was “attending programs designed for Attending programs designed for them 70.3 
them.” The other three response options Helping with program planning 11.4 
each had average participation of fewer Participating in program evaluations 17.4 
than 20 adults. Helping deliver services 11.9 


Data source: 21st CCLC Program Director Survey May- 
June 2016 


EQ5 Improvement and Accountability Processes 


How helpful to 21st CCLC programs were improvement/accountability processes? 


Figure 6 shows the large 
majority of program directors 
found both improvement and 
accountability process to be 
helpful, although the WVDE 
monitoring visits receive more 
moderately helpful and very 
helpful ratings than the 
continuous improvement 
process for after school 
(CIPAS) did. Six program 
directors indicated the CIPAS 
was not applicable and three 
that the WVDE monitoring 
process was not applicable, 
because they had not 
completed the process or had 
not received a visit. 
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= Not applicable m= Not very helpful Neutral 


= Moderately helpful = Very helpful 


Figure 6. Helpfulness of Improvement and Accountability 
Processes 

Data source: 2016 21st CCLC Program Director Survey May-June 

2016 


EQ6 Successes, Challenges, and Recommendations 


What do program directors view as their major successes, challenges, and recommendations 


for the future of the program? 
Successes 


Program improvement. About two thirds of program directors mentioned improvements in the 
variety and quality of their programming. Aside from the frequent mentions of homework help 
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and tutoring, and there were frequent mentions of literacy activities (there were no similar 
mentions of math activities). Programs mentioned having enrichment and recreational activities 
available for after students finished their homework, such as at one program, "gym games, art, 
recording studio, music mentor sessions, dance, computer based learning, x-box learning, 
nature learning lab, and gardening." Other programs mentioned robotics, cooking, hunter safety, 
"Girls Who Code," Legos activities, and others. 


One program described their success this way, "We continue to be able to provide a safe 
environment for students to complete homework, receive tutoring and academic enrichment, 
participate in recreation and have a snack." 


Family and community involvement. The multiple choice section of the survey showed that 
61% of programs directors indicated moderate to great success in the area of family and 
community involvement, compared with 39% in 2014-2015. Success narratives included the 
following examples: 


e Community engagement and sustainability 

o "“Agroup of about 15 with stakeholder interview([s] for another 30 took part in 
planning the renewal grant for this area." 

o "Ata youth summit teens identified the need to learn about work futures and over 
37 businesses responded by pledging their support for programing." 

o “We have increased our partnerships and solidified our presence in the 
community.” 

o “Project staff have attended additional community events to promote the 
program.” 

e Family and community events 

o “Greatly increased our community relations through our once a month family night. 
Which average 70 student and family participants.” 

o “At one site we started making invitations for the students to invite parents, 
grandparents, and community/school members, with these we have seen a large 
increase in participation in these events.” 

e Interagency collaboration, included working with housing authorities, health 
departments and nonprofits 


School system engagement/support. Seven programs reported increased collaboration with 
school systems. Examples included expansion to additional schools within districts, provision of 
office space by another school (Preston High School), and the schools helping more to promote 
the program. 


Working with university student volunteers was cited by five program directors. Colleges 
and universities mentioned included the University of Charleston, Berea College, Earlham 
College, West Virginia State University, Fairmont State University, Pierpont Technical College, 
and Blue Ridge Community and Technical College. The multiple choice portion of the survey, in 
a question about sources of volunteers, indicated that the slight majority who used volunteers 
from higher education gave those volunteers high ratings. 
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Discussion and 
Implications 


Staffing and staff development. Six program directors mentioned improvements in their 
staffing with, in two cases, benefits for students in making some staffing changes. Only one 
program counted the provision of ongoing, monthly professional development as one of their 
successes in their success narratives. 


Student participation. Seven mentioned improvements in student participation and 
attendance. 


Challenges 


Staffing and staff development. Thirteen program directors mentioned staffing challenges, 
including the need for training for staff and to increase staff retention, the need for additional 
staff to keep up with growing student participation and to fill openings created by staff turnover. 
They also mentioned difficulties in hiring substitutes and filling tutoring positions. 


Collaboration with regular-day teachers and administrators. County and school 
administrative staff can stand in the way of progress--two examples in one county. There was a 
lack of cooperation in extending transportation to a new site. At one school a principal refused to 
cooperate, making it necessary to work directly with individual teachers. However, athletic 
coaches at another school helped advocate for use of the programs with their athletes. 


Funding and sustainability. Eight program directors mentioned funding, including challenges 
making payroll, providing scholarships to attend afterschool and summer programs, finding 
administrative time to write grants, and obtaining long-term funding from partners. 


Improving family and community involvement continues to be a challenge for eight program 
directors. To address the issue some reported sending special invitations to targeted parents, 
re-establishing advisory councils, setting up special programs for fathers, expanding offerings, 
and conducting parent surveys. 


Student participation. Seven program directors reported difficulties with keeping students in 
the program and maintaining their consistent attendance. "...there is a revolving door of children 
entering and leaving the schools like most areas of West Virginia hit hard by the economic 
downturn." This includes one area, where there is high "turnover among families at particular 
housing developments." Solutions sought included developing "a valley wide solution so we can 
follow kids as they move from development to development." Athletic programs also pull 
students away from afterschool programs, especially in the spring. 


Discussion and Implications 


Program directors reported significant progress in several areas, including program 
improvement, school system engagement and support, working with university student 
volunteers, staffing and staff development, and student participation. Perhaps most notably, 
however, were the reports and ratings indicating more success with parent and community 
engagement. 
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Discussion and 
Implications 


Program directors also identified challenges they encountered in the area of staffing related to 
retaining and hiring staff and providing ongoing training. Sometimes uncooperative school 
administrators throw up obstacles to the kind of collaboration that needs to take place between 
regular-day teachers and 21st Century Community Learning Centers (21st CCLC) staff in order 
to provide appropriate services for students. Funding is a perennial problem, and some 
programs continue to struggle with family and community involvement. Seven programs 
struggled with student retention. This challenge was born out in average percentages of 
students who attended 30 days or more, which ranged from about 11% to nearly 90%. Perhaps 
there are things programs struggling with student retention can learn from those that are having 
more success. 


Program directors made the following recommendations: 


e Provide program directors with networking opportunities, site visits. There were 10 
mentions of the desire for networking opportunities among program directors for the 
purpose of sharing best practices. There were also numerous requests for more sharing 
of best practices during training, and a few calls for fewer sales presentations. 

e Improve reporting platforms and processes; use results for program 
improvement. There were far fewer requests related to improving online reporting 
systems than in previous years, and three mentions of recent improvements to some of 
the systems. Still there were a few calls for more streamlining and the reduction of 
redundancy in reporting. 


Improvements were made to the Program Director Survey in 2015-2016 in response to 
recommendations made in recent years. Also, due to relaxing of some federal requirements the 
Teacher Survey was greatly simplified, which seems to have resulted in a higher number of 
teacher responses. Still a detailed look at what data are collected and how those data are used 
for fulfilling federal requirements and program improvement could help further streamline the 
system. 
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Appendix A 
Detailed Description of Study Methods 


Methods Used to Address EQ1 


Descriptive statistics using West Virginia Department of Education (WVDE) 21% 
Century Community Learning Center (21S' CCLC) database 


The following statistics were derived using data from the WVDE 21*t CCLC database: 


e The number of students and their distribution by grade level 
e The average student attendance (dose strength) by program 


Data collected in this database were submitted by the site managers and program directors 
on a daily basis. 


Descriptive statistics using online 21S‘ CCLC Teacher Survey 


Based on responses to the Teacher Survey (see Appendix B), we calculated an effect size 
of the program on various behaviors by using a retrospective pre-post measure. Based on 
responses to the Teacher Survey and the WVDE 21% CCLC database, we determined if 
there was a relationship between reasons for referral and the length of time students spent 
in the program beyond 30 days. 


Population characteristics and sampling procedures 


The sample for the Teacher Survey started with the teachers of those students who 
participated in the 21st CCLC program for at least 30-days whose parents had given passive 
consent. Consent forms were given to parents at the time they registered their children; 
parents were instructed that if they agreed to have their children be part of the evaluation, 

no action was necessary. If they denied consent, they returned the signed form, which will 
be kept in program directors’ offices until April of each year, and then sent to the WVDE 
Office of Research, Accountability, and Data Governance. Denial of consent was logged into 
the 21st CCLC database maintained at WVDE, using a checkbox added to the system for 
the 2015-2016 school year and thereafter. 


Quasi-experimental study 


We conducted a quasi-experimental examination of existing student assessment data 
obtained from the West Virginia Education Information System (WVEIS) in mathematics and 
English/language arts (ELA) for students who participated in a 21st CCLC during the one- 
school-year study period compared with a matched group of students who were not known 
to have participated. 
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Population characteristics 


The treatment group consisted of students who participated in at least 30 days of 21st 
CCLC intervention as documented in the WVDE 21*' CCLC database. The comparison 
group was matched using propensity scores, matching a variety of demographic and 
performance covariates. Analyses were conducted to examine both within- and between- 
group differences in student achievement. 


Sampling procedures 


From attendance records submitted by site managers to the WVDE 21°! CCLC database 
(not the Teacher Survey dataset), we identified students who 


e were in Grade 4 or above 

e who had test records for both school years 

e who received scale scores for both mathematics and ELA 

e who had a complete set of demographic covariate variables to be used during 
matching 

e who were not retained from one year to the next. 


After removing students who do not meet all of these criteria, we were left with our final 
sample. 


We used propensity score matching (PSM) to select a matched comparison group for each 
grade level. This methodology used logistic regression to select a comparison group that 
closely matched the treatment group on a variety of observed covariates. First, a binary 
indicator showed whether or not each student in the state participated in 21st CCLC during 
the study school year. Group 1 was defined as the treatment group (those students who 
attended 30 or more days in a 21st CCLC) and Group 0 was the control group (those 
students who did not participate in 21st CCLC during the school year studied). We then 
derived conditional probabilities for each student by regressing the binary group 
membership variable on the following covariates: (a) prior academic achievement in both 
mathematics and ELA, (b) gender, (c) race/ethnicity, (d) free/reduced-price lunch eligibility, 
and (e) special education eligibility. Grade level was held constant by conducting matching 
independently within each grade. Thus, in this study the propensity score represents the 
predicted probability that a given student would attend 30 days of 21st CCLC based on this 
set of pre-intervention covariates. Finally, we used nearest-neighbor matching to select the 
most appropriate match for each 21st CCLC student. Verification analyses was conducted 
to check that this matching methodology identified an adequately balanced comparison 
group for hypothesis testing.® 


3 We used chi squared analyses to verify the two groups did not differ on categorical demographic 
variables. We also used independent samples t-tests to verify the two groups did not differ on prior 
academic achievement in ELA and mathematics. 
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Measures and covariates 


This portion of the study includes an examination of student achievement data from the 
West Virginia General Summative Assessment (WVGSA). We analyzed individual students’ 
scale scores, and gain scores in both mathematics and ELA. Gain scores were 
operationalized as the change in student scale scores from the previous school year to the 
study school year, with the expectation that students who participated in 21st CCLC during 
the study school year would experience differential gains when compared with similar 
students not known to have participated in 21st CCLC. 


Data collection methods 


All data for the quasi-experimental portion of this study was collected from two sources—the 
21st CCLC database and WVEIS general summative assessment records file, both 
maintained by the WVDE. 


Research design 


Two sets of analyses were run to answer the question, “Was mathematics and ELA 
achievement attained by 21st CCLC participants significantly different from that attained by 
nonparticipants ?” 


The first analysis used independent samples f tests (one per grade for Grades 4—11) to 
determine whether students in the treatment group scored significantly higher than students 
in the control group on WVGSA mathematics outcomes. Another set of t tests were run for 
ELA outcomes. 


The second analysis used a Pearson’s chi-square test to determine whether students in the 
treatment and control groups—at three programmatic levels, including elementary school 
(Grades 4 through 5), middle school (Grades 6 through 8), and high school (Grades 9 
through 11)—differed in making gains in their performance levels. The performance levels 
were on the following scale: 4 = exceeded the standard, 3 = met the standard, 2 = nearly 
met the standard, and 7 = has not met the standard. 


Methods Used to Address EQ2-EQ6 


The final evaluation questions were addressed using descriptive statistics and qualitative 
analysis of responses to the Online 21st CCLC Program Director Survey (see Appendix B, 
page 20 for screen shots of the questionnaire). Responses to multiple choice questions 
were tabulated and interpreted, and open-ended questions were analyzed using the RQDA 
qualitative data software application. 
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Appendix B 
Program Director and Teacher Survey Questionnaires 


Program Director Questionnaire 


Volunteer Programs 


If your program uses volunteers, which sources have you used, how many, and with what level of success? 


Number of Volunteers anaane al 
(none used) 

AmeriCorps (AmericCorps Promise Fellow, AmeriCorps VISTA, Citizen a | @ | 
Community Corps) Le 

Senior Corps (Retired and Senior Volunteers, Foster Grandparent Program) Zz - Koa 

Service Learning (K-12 students) 10 | @ | 

Service Learning (Higher Education students) 10 | @ | 

Local Businesses 37 | @ | 

Community Organizations (21 | @ | 

Faith Based Organizations [2 } | @ | 

Parents 21 ne 

Faculty Members 27 | | @ | 

Local Clubs (e.g. Kiwanis, Lions) oO 5 > Koa 

Other 1 ] | @ | 


Survey Closed on July 31, 2016 


Partnerships 
lf your program utilizes partners, please indicate for each of the following forms of support, how many partners do you have 


ngaged and at what level of success. 


(no partnership) 

a 57 O et ar awe 
acini is — + 
randee —_ =a ac a | 
rail | ——- 
Training ; Cs Ea sce 
Management _— — 
Evaluation 5 3 el (Ca ome 
other oo ° —<— 


Survey Closed on July 31, 2016 
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Professional Development—Counties and RESAs 


For each of the following topics, please rate the level of effectiveness of any COUNTY OR RESA training you have received 
during this program year. 


Please check the preferred format if 
more information is required on this 
topic from the COUNTY or RESA 
Professional Technical Information 
Development Assistance Resources 

Lj 


Programming 
Collaboration 
Communications/Marketing 
Staff Development 
Integrating Afterschool with 
the Regular School Day 
Project Management 
Federal/State 
Requirements 
Family Involvement 
Program Sustainability 
STEM/STEAM 
Program Evaluation 
Policy and Advocacy 
Other 


Professional Development—West Virginia Department of Education 


For each of the following topics, please rate the level of effectiveness of any WVDE training you have received during this 
program year. 


Please check the preferred format if 
more information is required on this 
topic from the WYDE 
Professional Technical Information 
Development Assistance Resources 

\ I} 
a is 


Programming 
Collaboration 
Communications/Marketing 
Staff Development 
Integrating Afterschool with 
the Regular School Day 
Project Management 
Federal/State 
Requirements 
Family Involvement 
Program Sustainability 
STEM/STEAM 
Program Evaluation 

Policy and Advocacy is 
Other © 
Professional Development—U.S. Department of Education 


For each of the following topics, please rate the level of effectiveness of any USDE training you have received during this 
rogram year. 


{ } { 


Please check the preferred format if 
more information is required on this 
topic from the USDE 
Professional Technical Information 
Development Assistance Resources 
Programming © LJ & 

Collaboration e 
Communications/Marketing © 
Staff Development © 
Integrating Afterschool with 
the Regular School Day 
Project Management * 
Federal/State 
Requirements 
Family Involvement 
Program Sustainability 
STEM/STEAM 
Program Evaluation 
Policy and Advocacy 
Other 
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Appendix B 


Technical Assistance 


Please indicate the level of helpfulness of each of the following forms of technical assistance you may have received. 


Email kk 

Phone Call/Conference Call = Sea o 
Webinar = 

Action Plan Feedback ES ey oe 
Peer Learning Teams SSS & 
Site Visit ES ea S 

CIPAS = We S 

Other Si So 


Parent/Community Involvement 


Parent and Community Involvement 


How successful has your program been in involving 
parents/guardians or other adult community members? 


How many adults have participated in your programs b ane 


Attending programs designed for them? 113 
Helping with program planning? 15 
Participating in program evaluations? 5 
Helping deliver services? (ios 


Continuous Improvement Process 


IPAS - Continuous Improvement Process for After School 


How helpful has the Continuous el ement Process for After School ee ae IPAS) been to your [ana =] 


Not applicable -| didn't 
complete the CIPAS sss [enenneunnn 


a isaac amis MEN SESE eee, “ESE 


Program Mentoingeeces Process 


Program Monitoring Process 


How helpful have the monitoring visits by WVDE staff to your site been this year? 
Not applicable - | didn't 
complete the CIPAS process 


My program was not visited 
by a WVDE monitor this year. 
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Teacher Survey 


21% CCLC Teacher Survey WVDE-EQSS-001 
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Appendix C 


Appendix C 
Informed Consent Forms 


Informed Consent of Parents/Guardians 


[Printed on WVDE letterhead] 
2015-2016 Evaluation of West Virginia’s 
21st Century Community Learning Center Program 
Parent/Guardian Informed Consent 


I understand that the afterschool program my child will attend will be evaluated by the 
West Virginia Department of Education (WVDE). The purpose of the evaluation study is to find 
out how well the program is working. What the WVDE learns from this study may help improve 
the program in the future. Later this school year, we would like to ask your child’s teacher about 
the amount of progress your child has made. Any information we would gather would be 
protected and your child would never be identified. The information provided would be 
combined with information from others, and reported as a group. 

Allowing your child to take part in this study in the way just described will put your child 
at no more risk than he or she would experience during any normal day. Although your child may 
not benefit directly by being part of the study, it is possible that because of what we learn, the 
program may improve to better meet his or her needs or the needs of other students. 

Neither you nor your child will receive any money or other reward for taking part in this 
study. Allowing your child to be part of the study is completely voluntary. If you decide not to 
allow your child to be part of it, there will be no penalties or loss of benefits to you or your child. 

To allow us to collect this information from your child’s teacher there is no action 
you need to take. Thank you! 

If you do NOT want your child to be part of the study, just fill in the information 
below and return this form to the afterschool program coordinator. 


LI Do NOT include my child in the evaluation study. 


Child’s name (please print): 


Parent/guardian signature: Date: 


Name of afterschool program: (to be filled in by program staff): 


For more information about the education program we are studying, you may contact Josh Asbury (304- 
872-6440, jmasbury@k12.wv.us) or Benitez Jackson (304-256-4712; bljackso@k12.wv.us). If you have 
questions about this evaluation study, you may contact Patricia Hammer (304-558-2546; 
phammer@k12.wv.us). This study has been reviewed and approved by the West Virginia Department of 
Education Institutional Review Board (IRB-WVDE-031)). If you want to know more about the review of 
this study, you may contact the WVDE IRB cochair, Andy Whisman (awhisman.k12.wv.us). 

Parents: Keep a copy of this form for your records. 
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Appendix C 


Informed Consent for Teacher Survey 


By filling out this survey, you are agreeing to take part in an evaluation study. The purpose 
of the study is to evaluate the effectiveness of the 218' Century Community Learning Center 
(21S CCLC) program. What we learn from this study may help improve this program or other 
afterschool programs in the future. Your participation in the study is limited to completing a 
brief survey about each of your students, which should not take more than 3—4 minutes for 
each student. You will be presented with a series of items and asked to indicate your 
responses by either checking off a rating. 


Taking part in this study will put you at no more risk than you would experience during any 
normal day. Although you may not benefit directly by taking part in the study, it is possible 
that because of what we learn, the program may improve to better meet your needs or the 
needs of students. Your responses to this survey will be protected and will never be 
revealed as coming from you. All responses will be combined and reported as a group. 


You will receive no monetary or other reward for taking part in this research study. Filling out 
the survey is completely voluntary. If you decide not to take part or to stop at any time, there 
will be no penalties or loss of benefits to you. For more information about the education 
program we are studying, you may contact Josh Asbury (304-872-6440, 
jmasbury@k12.wv.us) or Benitez Jackson (804-256-4712; bljackso@k12.wv.us). If you have 
questions about this evaluation study, you may contact Patricia Hammer (804-558-2546; 
phammer@k12.wv.us). This study has been reviewed and approved by the West Virginia 
Department of Education Institutional Review Board (IRB-WVDE-031)). If you want to know 
more about the review of this study, you may contact the WVDE IRB cochair, Andy 
Whisman (awhisman.k12.wv.us). 


Thank you for taking part in this important effort. 
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Appendix C 


Informed Consent for Program Director Survey 


By filling out this survey, you are agreeing to take part in an evaluation study. The purpose 
of the study is to find out how well various aspects of the 21st Century Community Learning 
Center program are working and to gather information that can be used to guide the 
program in the future. What we learn from this study may help improve the program or other 
education programs. To be part of the study, all you need to do is complete a survey by 
checking off your answers to the questions. Some questions ask you to write an answer ina 
text box. Filling out the survey should not take more than 30 minutes. 


Taking part in this study will put you at no more risk than you would experience during any 
normal day. Although you may not benefit directly by taking part in the study, it is possible 
that because of what we learn, the program may improve to better meet your needs or the 
needs of students. Your responses to this survey will be protected and will never be 
revealed as coming from you. Your responses will be combined with responses from others, 
and reported as a group. 


You will receive no money or other reward for taking part in this research study. Filling out 
the survey is completely voluntary. If you decide not to take part or to stop at any time, there 
will be no penalties or loss of benefits to you. For more information about the education 
program we are studying, you may contact Josh Asbury (304-872-6440, 
jmasbury@k12.wv.us) or Benitez Jackson (804-256-4712; bljackso@k12.wv.us). If you have 
questions about this evaluation study, you may contact Patricia Hammer (804-558-2546; 
phammer@k12.wv.us). This study has been reviewed and approved by the West Virginia 
Department of Education Institutional Review Board (IRB-WVDE-031)). If you want to know 
more about the review of this study, you may contact the WVDE IRB cochair, Andy 
Whisman (awhisman.k12.wv.us). 
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